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Abstract:  Inevitable  camera  motion  during  exposure  does  not  augur  well  for  free-hand 
photography.  Distortions  introduced  in  images  can  be  of  different  types  and  mainly  depend  on 
the  structure  of  the  scene,  the  nature  of  camera  motion,  and  the  shutter  mechanism  of  the  camera. 
Most  present-day  imaging  devices  are  equipped  with  CMOS  sensors  and  motion  blur  is  a 
common  artifact  in  handheld  cameras.  Because  CMOS  sensors  mostly  employ  a  rolling  shutter 
(RS),  the  key  challenges  of  registration,  change  detection,  and  motion  deblurring  problem  take 
on  new  dimensions.  Although  few  works  have  recently  addressed  these  problems,  they  suffer 
from  many  constraints  including  high  computational  cost,  need  for  precise  sensor  information, 
and  inability  to  deal  with  wide-angle  systems  (which  most  cell-phone  and  drone  cameras  are) 
and  irregular  camera  trajectory. 

In  the  first  part  of  this  work,  we  attend  to  the  coalesced  effect  of  rolling  shutter  and  motion  blur 
that  occurs  frequently  in  moving  CMOS  cameras.  We  first  model  a  general  image  formation 
framework  for  a  3D  scene  following  a  layered  approach  in  the  presence  of  rolling  shutter  and 
motion  blur.  We  then  develop  an  algorithm  which  performs  layered  registration  to  detect 
changes.  This  algorithm  includes  an  optimization  problem  that  leverages  the  sparsity  of  the 
camera  trajectory  in  the  pose  space  and  the  sparsity  of  changes  in  the  spatial  domain.  In  the 
second  part  of  the  work,  we  address  the  problem  of  registering  images  taken  from  global  shutter 
and  rolling  shutter  (RS)  cameras  and  reveal  the  constraints  on  camera  motion  that  admit 
registration,  change  detection,  and  rectification.  Our  analysis  encompasses  degradations  arising 
from  camera  motion  during  exposure  and  differences  in  shutter  mechanisms.  We  also  investigate 
conditions  under  which  camera  motions  causing  distortions  in  reference  and  target  image  can  be 
decoupled  to  yield  the  underlying  latent  image  through  RS  rectification.  In  the  final  part  of  the 
work,  we  propose  a  model  for  RS  blind  motion  deblurring  that  mitigates  many  of  the  associated 
issues  significantly.  Comprehensive  comparisons  with  state-of-the  art  methods  reveal  that  our 
approach  not  only  exhibits  significant  computational  gains  and  unconstrained  functionality  but 
also  leads  to  improved  performance. 

The  three  publications  related  to  these  works  contain  detailed  and  complete  descriptions  of  the 
proposed  methodologies  for  registration,  change  detection,  and  deblurring  in  rolling-shutter 
images.  We  request  the  reader  to  refer  to  these  papers  which  have  been  provided  as  attachments. 
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